P-values

What does a p-value mean?

What does a p-value mean?

What does a p-value mean?

• In the broadest sense, p-values indicate the probability of getting a result >, <, or absolutely greater than as the one in your sample if the null hypothesis is true.

  • Note, this is a statement about the probability of data given a model, not a statement about the probability of a model.

• In linear regression, p-values are typically calculated based on the normal distribution

P-values and Type I error

.05 level means that around 1 in 20 of results would be false positives

What does a p-value not mean?

• \(P(t | h_0) \neq P(h_1)\) P values are not the probability of the alternative

P-values: caveats

• P1. p-values can indicate how incompatible the data are with a specified statistical model.
• P2. p-values do not measure the probability that the studied hypothesis is true, or the probability that the data were produced by random chance alone.
• P3. Scientific conclusions and business or policy decisions should not be based only on whether a p-value passes a specific threshold.
• P4. Proper inference requires full reporting and transparency.
• P5. A p-value, or statistical significance, does not measure the size of an effect or the importance of a result.
• P6. By itself, a p-value does not provide a good measure of evidence regarding a model or hypothesis

From: McShane, B. B., & Gal, D. (2017). Statistical significance and the dichotomization of evidence. Journal of the American Statistical Association, 112(519), 885-895.

Problems with P-values

• Often misinterpreted, even by experts

• They’re continuous, but they tend to be treated as binary and conflated with importance

• In large samples, they’re hard to read because they get infinitesimally small

Confidence intervals

What do confidence intervals mean?

In repeated sampling, 95% of the 95% confidence intervals will contain the correct value of \(\beta\)

Confidence Intervals

• Its misleading to say there’s a 95% chance that the population \(\beta\) falls between the ci boundaries. (frequentist stats assumes that \(\beta\) is fixed, so it doesn’t have a “chance” of doing anything.)

• Better to say “I’m 95% certain this range contains the true value”

Confidence Intervals

• Pro: P.values and Confidence Intervals are based on the same basic assumptions, but CIs can be more informative about uncertainty and don’t lend themselves to the same dichotomous interpretation

• Pro: Automatically scaled to the quantity of interest, so harder to confuse with a measure of substantive importance

• Pro: can be used for equivalence testing (we’ll come back to this)

• Con: also commonly misinterpreted!

Alternatives: S-Values

Alternatives: S-Values

• Pro: Still based on the same assumptions as a p-value, its just a transformation of the p.value that might be more intuitive for people.

• Pro: easy calculation (for a computer, at least)

• Con: Not widely adopted. All of this stuff relies on convention.

Bayesian methods

• Bayesian methods potentially allow us to do the thing we wish we could do with p-values: quantify hypothesis probabilities

• Downside is that it sort of requires everyone to adopt a totally different philosophical approach to probabilities.

Bayes Factors Approximation

Bayes Factors measure the weight of evidence in favor of one hypothesis relative to another.

Wagenmakers (2007) suggests a frequentist approximation of a Bayes Factor using the Bayesian Information Criterion.

ff<-read_csv("https://vincentarelbundock.github.io/Rdatasets/csv/causaldata/gov_transfers.csv")

model_0<-lm(Support ~  Income_Centered + Education + Age, data=ff)

model_1<-lm(Support ~  Participation + Income_Centered + Education + Age, data=ff)





exp(-BIC(model_1)/2)/exp(-BIC(model_0)/2)

[1] 7.509412

Numbers greater than 1 indicate evidence in favor of model 1, numbers less than 1 indicate evidence against model 1.

Comparing Models

• If I have multiple models, how do I tell whether one is better than the other?

“Good” models can be bad predictors

R-squared

R-squared can be used for model comparison, but it doesn’t tell you a model is a good predictor.

Comparing models

• Make a fair comparison: our “restricted” model \(Y_i = \bar{Y} + e_i\)

Comparing models

Predicting Democratic voteshare using the 2016 share

Comparing models

\[ F = \frac{(RSS_{restricted} - RSS_{full})}{df_{restricted} - df_{full}} \div \frac{RSS_{full}}{df_{full}} \]

model_0 <- lm(demshare~ 1, data=voteshares)

model_1 <-lm(demshare ~ 1 + demshare_2016, data=voteshares)

anova(model_0, model_1)

Comparing models

We can continue to include additional covariates in the model. But the “fair comparison” here is probably the same model -1 covariate (rather than comparing to the null model)

BIC

The F-test requires nested models, but the BIC and AIC allow comparisons across non-nested forms

\[{\displaystyle \mathrm {BIC} =n\ln(RSS/n)+k\ln(n)\ }\]

BIC(model_1)

[1] -284.4724

BIC(model_2)

[1] -280.919

Out of sample prediction

• Basic method: set aside some observations, create model on the training set and then predict the test set

• For small samples, K-fold or LOOCV are preferable: leave out one, predict, and then do it again. Then average the metric.

library(caret)
# Define training control
train.control <- trainControl(method = "LOOCV")
# Train the model
cv_1 <- train(demshare ~ demshare_2016, data = voteshares, 
               method = "lm",
               trControl = train.control)

Out of sample prediction

Linear Regression 

51 samples
 1 predictor

No pre-processing
Resampling: Leave-One-Out Cross-Validation 
Summary of sample sizes: 50, 50, 50, 50, 50, 50, ... 
Resampling results:

  RMSE        Rsquared   MAE       
  0.01380351  0.9871893  0.01096933

Tuning parameter 'intercept' was held constant at a value of TRUE

Out of sample prediction

…but is this the right metric?

Arguing for a null

Conventional hypothesis testing “stacks the deck” in favor of finding no relationship, and failure to reject the null could simply be a function of small sample sizes.

Still, we often want to dismiss a claim:

• Can citizen pressure campaigns make states more effective?

• Do motor voter laws improve turnout?

• Are some people actually psychic?

TOST and equivalence testing

“Two one-sided tests”

Step 1: Set a minimum effect size (\(\Delta\)) worth caring about

Step 2: Conduct one-sided t-test for:

• H01: \(\Delta \leq −Δ_l\)

• H02: \(\Delta \geq Δ_u\)

If both of these tests are rejected, then we can conclude the observed effect is smaller than the minimum effect size.

TOST and equivalence testing

TOST and equivalence testing

Moery and Calin-Jageman’s replication results look like this:

TOST and equivalence testing

The estimated effect: 5.25 - 5.22 = 0.03

Minimal effect size \(\Delta = 0.43\)

Negligible Effects (Rainey)

Null effects and Duverger’s law

Rainey:

• “Hypothesis 1: Increasing the effective number of ethnic groups from the 10th percentile (1.06) to the 90th percentile (2.48) will not lead to a substantively meaningful change in the effective number of political parties when the district magnitude is one.”

• “Hypothesis 2: Increasing the district magnitude from one to seven will not lead to a substantively meaningful change in the effective number of political parties when the effective number of ethnic groups is one.”

• Substantively meaningful effect: changing the effective number of parties by 0.62 (where does he get that?)